An Approximate Dynamic Programming Algorithm for Large-Scale Fleet Management: A Case Application

We address the problem of developing a model to simulate at a high level of detail the movements of over 6,000 drivers for Schneider National, the largest truckload motor carrier in the United States. The goal of the model is not to obtain a better solution but rather to closely match a number of operational statistics. In addition to the need to capture a wide range of operational issues, the model had to match the performance of a highly skilled group of dispatchers, while also returning the marginal value of drivers domiciled at different locations. These requirements dictated that it was not enough to optimize at each point in time (something that could be easily handled by a simulation model) but also over time. The project required bringing together years of research in approximate dynamic programming, merging math programming with machine learning, to solve dynamic programs with extremely highdimensional state variables. The result was a model that closely calibrated against real-world operations, and produced accurate estimates of the marginal value of 300 different types of drivers. In 2003, Schneider National, the largest truckload motor carrier in the United States, contracted with CASTLE Laboratory at Princeton University to develop a model that would simulate its long-haul truckload operations to perform analyses to answer questions ranging from the size and mix of its driver pool to questions about valuing contracts and getting drivers home. The requirements for the simulator seemed quite simple: it had to capture the dynamics of the real problem, producing behaviors that closely matched corporate performance along several dimensions, and it had to provide estimates of the marginal value of different types of drivers. If the model accurately matched historical performance, the company would be able to use the system to test changes in the mix of drivers, the mix of freight and other operating policies. The major challenge we faced was that these requirements required that we do much more than develop a classical simulator. It was not enough to optimize decisions (in the form of matching drivers to loads) at a point in time. It had to optimize decisions over time so that decisions took into account downstream impacts. Formulating the problem as a deterministic, time-space network problem was both computationally intractable (the problem is huge) and too limiting (we needed to model different forms of uncertainty, as well as a high degree of realism that was beyond the capabilities of classical math programs). Classical techniques from Markov decision processes applied to this setting are limited to problems with only a small number of identical trucks moving between a few locations (see Powell (1988) or Kleywegt et al. (2004)). Our problem involved modeling thousands of drivers at a high level of detail. We solved the problem using approximate dynamic programming, but even classical ADP techniques (Bertsekas & Tsitsiklis (1996), Sutton & Barto (1998)) would not handle the requirements of this project. Three years of development produced a model that closely matches a range of historical metrics. Achieving this goal required drawing on the research of three Ph.D. dissertations (Spivey (2001), Marar (2002), and George (2005)), and depended on the extensive participation of the sponsor to produce a model that accurately simulated operations. The model is able to handle a host of engineering details to allow the sponsor to run a broad range of simulations. To establish credibility, the model had to match historical performance of a dozen major operating statistics. Two of particular importance to our presentation